Sargent was born in Bath Villas, Ashford in Kent England to a working-class, but musical family. He was brought up in Stamford, Lincolnshire where he won a scholarship to Stamford School. At the age of fourteen, he accompanied rehearsals for an amateur production of The Gondoliers at Stamford. He earned his diploma from the Royal College of Organists at age sixteen. After a brief service in the army, Sargent worked first as an organist at Melton Mowbray Parish Church, Leicestershire.